Papers with interactive evaluation framework
Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Task-oriented dialogue systems are drawing more attention in recent studies . current evaluation methods use annotated utterances in multi-turn dialogue sessions . |
| Approach: | They propose an interactive evaluation framework for task-oriented dialogue (TOD) they build a goal-oriented user simulator based on pre-trained models and use it to generate dialogues . |
| Outcome: | The proposed evaluation framework achieves 98% inform and success rates in the multi-woz dataset and the proposed scores measure the response quality besides the inform and successful rates. |
Strong Reasoning Isn’t Enough: Evaluating Evidence Elicitation in Interactive Diagnosis (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of medical consultation are static or outcome-centric, neglecting the evidence-gathering process. |
| Approach: | They propose an interactive evaluation framework that explicitly models the consultation process using a simulated patient and a measurement module grounded in atomic evidences. |
| Outcome: | The proposed evaluation framework outperforms baseline evaluation methods in medical consultation settings. |